DOCUMENT RESUME 



ED 367 684 



TM 021 134 



AUTHOR 
TITLE 



INSTITUTION 



SPONS AGENCY 

PUB DATE 
CONTRACT 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



Rothman, Robert 
Assessment Ques 
the 1993 CRESST 
September 12-14 
California Univ 
o£ Evaluation* ; 
Standards , and 
0££ice o£ Educa 
Washington, DC. 
94 

R117G10027 
21p. 

Collected Works 



tions: Equity Answers. Proceedings o£ 
Conference (Los Angeles, California, 

, 1993). Evaluation Consent. 

Los Angeles. Center for the Study 
Center for Research on Evaluation, 

Student Testing, Los Angeles, CA. 

tional Research and Improvement (ED), 



Conference Proceedings (021) 



IDENTIFIERS 



MFOl/PCOl Plus Postage. 
Access to Education; ^Conferences; Cost 
Effectiveness; Cultural Awareness; Cultural 
Differences; Definitions; ^Educational Assessment; 
Educational Change; Educational Objectives; 
Elementary Secondary Education; ^Equal Education; 
Outcomes of Education; Portfolios (Background 
Materials); Racial Differences; ^Standards; ^^Test 
Bias; Test Construction; Testing; Testing Programs; 
Test Interpretation 

Alternative Assessment; ^^Authentic Assessment; Large 
Scale Programs; ^'^Performance Based Evaluation; Re£orm 
Efforts 



ABSTRACT 

Focusing on one of the critical questions in the 
shift to new forms of assessment, researchers, policymakers, and 
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Zero in II in on one of the most 
critical issues in the shift to 
new forms of assessment, more 
than 200 researchers, policy mak- 
ers, and teachers gathered at the 
UCLA campus September 12-14, 
1993 to discuss questions sur- 
rounding equity. 

Meeting at the 1993 CRESST 
conference, entided "Assessment 
Questions: Equit\' Answers,'' par- 
ticipants outlined many of the 
concerns associated with the topic 
and offered ^ome possible solu- 
tions. 

Cl^SST Criteria of Fairness 

CRESST Co-director Robert 
Linn said in opening remarks that 
equit\' is at the center of debates 
over standards and assessments. 
Fairness is one of the most critical 
of the criteria developed by 
CRESST to evaluate new assess- 
ments, he noted. At the same 



time, he said, the report of the 
National Council on Education 
Standards and Testing and the 
Clinton Administration's Goals 
2000 legislation have put equit\' 
at the top of the agenda in the 
federal and state governments. 

"States and the national effort 
are focusing heavily on trying to 
establish ambitious content stan- 
dards," Linn said. "And assess- 
ment has been central to all the 
work put forward in curriculum 
reform. This has led to demands 
for high standards of student per- 
formance assessed by new assess- 
ments congruent with the con- 
tent standards."^ 

"If you put the two together," 
Linn said, "that leads immedi- 
ately to concerns about equity: 
what it means to give students a 
fair shot [at meeting the stan- 
dards], especially if there are high 
stakes attached.'* 
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Dean Theodore R. Mitchell of 
the UCLA Graduate School of 
Education said thv. current de- 
bate represents a "unique histori- 
cal moment." "For the first time," 
he said, "Americans are discuss- 
ing both equit\' and excellence, 
taking into account both inputs 
and outcomes. VVc are at the 
threshold of a breakthrough," 
Dean Mitchell said. 

But CRESST Associate Direc- 
tor Joan Herman cautioned that 
researchers do not yet have all the 
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answers to enable us to walk 
through that door. "Most don't 
agree on what the central ques- 
tions are," she said. 

Defining Equity* 

While focusing their attention 
on the role of assessment, many 
agreed that the issue of equit\' 
involves the education system as 
a whole. "If we can adequately 
teach all youngsters, we don't 
need to be as concerned with 
equitx-,"' said Edmund W. Gor- 
don, CRESST/City Universit>^ 
of New York and Yale University. 

"However,"' Gordon added, 
''that fact does not let those in- 
volved with assessments otr the 
hook. Assessments themselves can 
be made more equitable," he said, 
"and assessments can help make 
inputs more equitable." 



To attain equity... 
opportunities for dem- 
onstrating compe- 
tency must be diverse. 



But while agreeing on equit\' as 
a goal, researchers offered differ- 
ing perspectives on how to define 
the concept and how it might be 
achieved. To Gordon, equit\' is 
not equivalent to equality, but 
rather, sufficiency. To attain eq- 



uity, resources must be distrib- 
uted sufficiently and opportuni- 
ties for demonstrating compe- 
tency must be diverse. '^You may 
expose ail persons to the same 
standard," Gordon said, "but if 
the manner in which the standard 
is presented isn't one that matches 
the characteristics of each per- 
son, one cannot assen that it has 
been presented equitably." 

Gordon also laid down a chal- 
lenge to the assessment commu- 
nity and outlined three ways to 
make assessment more equitable. 

First, make better use of the 
information provided from assess- 
ment to allocate resources equita- 
bly. Second, develop new instru- 
ments and procedures to tap stu- 
dents' atfective traits, not j ust their 
cognitive skills. And third, con- 
duct research and development 
work to build on what is known 
about pluralism. Although pon- 
folios appear promising as a way 
of assessing students' abilities 
through diverse ways, Gordon 
warned against latching onto port- 
folios as the solution to ever\' 
problem. 

Gordon also cautioned that the 
problem of inequit)^ is a problem 
of the larger society outside 
school. As long as society contin- 
ues to reward uir-^ers and to 
screen out people '"not like us," 



inequities wil! remain regardless 
of what happens to assessments. 

Lauren B. Resnick, CRESST/ 
LRDC, University of Pittsburgh, 
argued that assessment can play a 
major role in alleviating inequi- 
ties. Like Gordon, Resnick said 
that the real issue is learning, and 
she said that changing assessment 
is vital to creating opportunity to 
learn for all Americans. 



"...equity ...is the 
right to achieve at lev- 
els sufficient to par- 
ticipate productively, 
and in a rewarding 
way, economically 
and civically." 



Resnick argued that we are 
heading down a revolutionary 
path, at the end of which all stu- 
dents will have a ri^fht to achieve. 
"What we mean by equity," she 
said, "is the right to achieve at 
levels sufficient to participate pro- 
ductively, and in a rewarding way, 
economically r.nd civically." 

Reaching such equit\% she said, 
will require defining the level of 
achievement all children must at- 
tain, a process that is now under 
way through the development of 
national standards. In addition, it 
will require holding ourselves re- 
sponsible for providing the op- 
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portunity for all students lo 
achieve to the desired level. Al- 
though the Hffhtio achieve could 
eventually become a legal stan- 
dard that would enable young 
people to demand the opportuni- 
ties to achieve, in the meantime 
this rijiht could scr\-c as a moral 
obligation for society. 

Lorraine McDonnell, CRESST/ 
University' of California, Santa 
Barbara, argued that attaining 
equit\* is a political, not a moral, 
challenge. She noted that Ameri- 
cans have long supported what 
she called procedural equity^ or a 
process that ensures that evcr\'- 
one has access to valued goods. 
But substantive equity, or equal 
results, has never enjoyed public 
support, in part because it de- 
mands redistribution of resources. 
Achieving that t\'pc of cquit\% 
McDonnell argued, demands ap- 
pealing to voters' self-interest for 
a better society in which young 
people are better educated. "One 
hopes for altruism, but it's hard to 
build a political majorit}' that way,*' 
she said. 

Similarly, Jeannie Oakes of the 
UCL\ Graduate School of Edu- 
cation also doubted that moral 
suasion would be sufficient to 
ensure equit\'. Many schools, she 
said, lack the human and material 
resources needed to create the 
conditions to provide high levels 



of instruction for all students, 
particularly low-income, minor- 
it}' students. What is needed arc 
opportunity-to-learn standards 
that would dctine a tair share of 
resources for schools. But that in 
itself may not be enough, Oakes 
added, because it is "astonishing 
howquicklygocxi data disappear.*' 
Instead, she said, schools should 
make the data-collection process a 
part of die improvement process. 

"My best guess at this point," 
Oakes said, "is that whatever pro- 
cess we engage in ^ov the regular 
business of collecting informa- 
tion and feeding it back into our 
systems probably better look 
prett\' much like the complex kind 
of teaching and learning we're 
hoping for in schools. Ideally, the 
process of assessing the qualit}* of 
students' opportunities will be- 
come indistinguishable from the 
very effort to create and improve 
those opportunities." 

Evaluating the Fairness of As- 
sessments 

Looking specifically at the issue 
of fairness and assessment, 
CRESST Co-director Robert 
Linn noted that the traditional 
methods of evaluating whether 
tests are fair — impact analyses, 
which measure differences in 
group performances; sensitivit\^ re- 



4 



views, which examine test con- 
tent with an eye toward eliminat- 
ing offensive or stereotyping ma- 
terial; and statistical analyses — 
are inadequate for use with per- 
formance assessments. 



Math assessments that 
demand substantial 
linguistic abiUty may 
be unfair to those lack- 
ing in that skill. 



Linn proposed additional factors 
that must be considered to deter- 
mine if new assessments are fair. 
These new factors include: 

• The intent of the measure, 
or the extent to which as- 
sessments measure ancillar\* 
skills that might provide an 
advantage to a particular 
group. As an example. Linn 
noted that math assess- 
ments that demand a sub- 
stantial amount of linguis- 
tic abil?t\' could prove a dis- 
advantage to those with 
math skills who lack read- 
ing and writing abilit\'. 

• Comparabilit\\ or the ex- 
tent to which the assess- 
ments allow variabilit}' in 
format or scoring. 
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• Choice of task. By allowdng 
students to choose their 
tasks, such as reading a 
book, \vc may be pro\iding 
ail advantage to those who 
are already familiarwith the 
books the\' choose. 

• Delivers' standards. To as- 
sure equit\', one must look 
not only at assessments, but 
theprON'ision of the instruc- 
tional experiences to stu- 
dents. 

Leigh Burstein, CRESST/ 
UCL\, also said that examining 
students' opportunity to learn is 
essential to evaluate the tairness 
f new assessments. ""You can't 
measure achievement without 
knowing the instructional condi- 
tions under which achievement 
occurs/' he said. Rut he cautioned 
that studies of opportunities to 
learn ha\'e thus far only taken 
place in low-stakes environments, 
not in situations where schools 
were held accountable for provid- 
ing such opportunities. 

Burstein pointed out t^vo ways 
in which students' opportunities 
could have an effect on their per- 
formance on alternative forms of 
assessments. First is the students' 
own experiences. Students accus- 
tomed to multiple- choice tests 



may balk at assessments that de- 
mand that they w^ritc responses, 
particularly in subjects other than 
English, He pointed out that in 
the 1990 National Assessment of 
Educational Progress, many stu- 
dents simply skipped over open- 
ended items, and there were eth- 
nic group diftcrences in the omit 
rates. 

In addition to the variations in 
student experiences, difterences 
in teachers' experiences can also 
affect students' performance, 
Burstein said. In reform environ- 
ments, urban teachers tend to have 
fewer chances to participate in 
developing and scoring new as- 
sessments than do teachers from 
suburban and rural areas, since it 
costs districts to send teachers to 
such sessions. 

Burstein described tvvo studies 
designed to examine students' cur- 
ricular experiences. The first looks 
at classroom "artifacts" — text- 
books, logs of daily acti\ities, 
homework assignments, in-class 
quizzes, and major assignments — 
and compares the findings with 
sur\'eys of teachers that attempt 
to get at students' learning op- 
portunities. The second, con- 
ducted as part of die California 
Learning Assessment System, asks 
students and teachers if they have 
done problems in their mathemat- 



ics classes like the open-ended 
ones on the assessment. 



Performance-based 
assessments use a small 
number of tasks that 
may favor one group 
over another. 



H.D. Hoover of the Univcrsit}' 
of Iowa, however, said it is un- 
clear whether differences in group 
performance on performance as* 
sessments reflect differences in 
curricular experiences or test bias. 
Hi said that, unlike multiple- 
choice tests, which can include a 
wide range of questions, pertbr- 
mance-based assessments use a 
small number of tasks that may 
favor one group o\'er another. A 
certain reading passage and its 
corresponding questions may ap- 
peal more to a student who has an 
inherent interest in the subject 
matter of that passage. 

**For fairness, you need a diver- 
si t\' of content and contexts," said 
Hoover. "Those of us who build 
standardized tests — that's what we 
do. We ask lots of questions, and 
balance questions." 

Data From Large-Scale Assess- 
ment Programs 

Whether because of differences 
in opportunitN' to learn or because 
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of the assessments themselves, 
the gaps between advantaged and 
disadvantaged students are not 
closing as schools shift to new 
forms of assessment. In fact, re- 
searchers who have studied new 
large-scale assessment programs 
have found that the gaps may be 
widening. 

Daniel Koretz, CRESST/ 
RAND, said that some teachers 
in Vermont agree that that state's 
pioneering portfolio assessment 
program improved education for 
traditionally low-performing stu- 
dents. However, he noted, such 
improvements were erratic. 



The use of portfolio 
assessments requires 
teachers to adapt to 
everchanging curricu- 
lar and instructional 
demands. 



"We see sign after sign after 
sign that teachers van' enor- 
mously in response to perfor- 
mance assessment,"" Koretz said. 
"They vary in how quickly they 
adapt to demands, and to what is 
expected of them.'" 

"Moreover,"" Koretz said, "the 
program itself of course has done 
nothing to alienate the condi- 
tions that have plagued low-per- 
forming students, such as pov- 



err'."" He concluded that improv- 
ing the level of student pertbr- 
mance through the use of portfo- 
lios is a "hard, arduous task" that 
will require money and an infra- 
structure to enable teachers to 
adapt to changing curricular and 
instructional demands. 

Likewise, CRESST partner 
Mar\^ Lee Smith from Arizona 
State Universit}^ found that the 
first year of implementation of 
Arizona"s performance-based as- 
sessment program failed to ad- 
dress the disparities between low- 
performing and high-performing 
schools. Although teachers had 
believed that the program would 
be a low- stakes exercise that would 
enable tliem to improve instruc- 
tion, in part through developing 
and scoring the assessment, in 
practice the Arizona Student As- 
sessment Program (ASAP), 
proved quite different. The scor- 
ing was done by a commercial 
publisher, not teachers, and the 
stakes went up when newspapers 
ranked school districts according 
to test scores. The state, more- 
over, proNided litde professional 
development to boost the capaci- 
ties of schools. 

Whether the ASAP program will 
eventually narrow the gaps in 
school performance is difficult to 
predict. 



We have seen teach- 
ers break an integrated 
unit into bits, and 
teach toward mastery 
of the bits. 



It is not yet clear,"" said Smith, 
''how ASAP will affect instruc- 
tion, because or in spite of its 
high-stakes function, although 
there are already foreshadowings. 
It is predictable that districts with 
adequate resources will do what is 
necessar)' to raise low scores. 
Whether they will take the high 
road — undertaking the time-con- 
suming and expensive professional 
and curriculum development 
work necessary to teach toward 
ambitious standards and a think- 
ing curriculum — or the low 
road — finding the tricks to inflate 
scores — remains to be seen. At 
this point, we have already docu- 
mented such actinties as teachers 
focusing pupils' attention on those 
parts of the assessment that will 
be scored. We have also seen 
instances of what we call dis-inU- 
jjratinff^ in which teachers who 
lack a thorough understanding of 
constructivist teaching take what 
was designed to be an integrated 
unit, break it into bits, and teach 
toward mastery of the bits."" 
As with the large-scale pro- 
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grams, Lorric A. Shepard, 
CRESST/Universin'ot'Colorado 
at Boulder, said that in classroom 
assessments, gains in students' 
abiiit\' to perform well on perfor- 
mance tasks come slowly and un- 
evenly. Drawing t'rom her 
CRESST research involving third 
graders in three Colorado schools, 
Shepard cited a problem that asked 
the students to complete a table 
to determine how many pitchers 
(of 4 cups each) it would take to 
have enough cups for all the stu- 
dents in the class. There were no 
diftcrences in performance be- 
tween Anglo and Latino students, 
but initially very few students in 
either group could even attempt 
the problem, hi the second year 
of the program, the majority of 
students could complete the table 
and wrote more to explain their 
answers than they did the first 
year, Shepard explained. But 
many still had a long way to go. 
"Ajiyone who thinks this can be 
put in place in a year or two is 
probably crazy,'" she said. 

But Jennifer Han^ey, a teacher 
at Cherry Drive Elementary 
School in Thornton, Colorado, 
who is part of the CBJiSST/Uni- 
versity of Colorado at Boulder 
study, said even small gains are 
valuable. By using a variet\' of 
alternative assessment methods. 



Har\xy now kjiows her students 
better than she ever did and she 
has evidence of their progress that 
she can show to parents. She also 
cited the case of one of her stu- 
dents, Jeff. 

"I had given him a runnmg 
record in October," said Han'ey; 
'*he read about a page and a half 
in 10 minutes. At the end of that 
time, I couldn't bear to watch his 
face anymore, it was so painful. 
So we stopped,,.. [He read] the 
exact same page in January. And 
he blew right through it. He did 
very well. Again, he was third 
grade, it was a preprimer, but 
that was progress. And I showed 
it to him and said, *Look what 
youVe done,' And he beamed 
and said, Tm getting better, 
aren't I?' That's where it's at, for 
me," 



Ideally, policy makers 
should weigh the costs 
and benefits of a pro- 
posed reform before 
implementation. 



Costs of Performance Assess- 
ment 

The benefits of performance 
assessment should also have a bear- 
ing on policy makers' decisions 
about whether to implement 



them, since policy makers should 
ideally weigh the costs and ben- 
efits of a proposed reform. But 
quantit\'ing the potential benefits 
has proved elusive, noted David 
Monk of Cornell UniversitN', and 
as a result, most of the discussion 
of costs has focused solely on ex- 
penditures. 

"You can't talk about cost with- 
out dealing with the benefit side 
of the equation," Monk said, 
"'That's a problem with perfor- 
mance assessment. The simple fact 
is, we don't know very much about 
what performance assessment pro- 
duces, or what kind of levels of 
resources are required for this to 
take place. In the absence of that 
kind of knowledge, you're in a bit 
of a dilemma tr\'ing to carr\' out a 
cost analysis t hat's more than an 
expenditiiio analysis," 

Monk said that an analysis he 
conducted for the New Standards 
Project produced a range of cost 
estimates for performance assess- 
ments, depending on the extent 
to which the assessment is added 
on to existing programs and the 
extent to which every student is 
tested. In the worst case scenario, 
in which the assessment was an 
addition to existing programs and 
ever)' student in three grades was 
tested, the assessment cost $97.4 
million oi $29 per pupil for a 
large state (Texas), S27.9 million 
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or $28 per pupil For a medium- 
sized state (Virginia), and S3. 5 
million or S37 per pupil for a 
small state (Vermont ). In all cases, 
a little less than a Fourth oF the 
expenditures went toward staff 
development, and the total ex- 
penditures amounted to between 
0.6 percent and 0.7 percent oF 
each state's education budget. 

Fritz Mulhauser oF the U.S. 
General Accounting OFfice 
(GAO), which conducted an 
analysis oFthe cost oF testing For 
Congress, said that the GAO study 
came up with an estimate for a 
proposed national test that was 
Far lower than previous esti- 
mates — From S42 million For a 
multiple-choice test and little 
added time to $209 million For a 
test including short perFormance- 
based questions and 30 minutes 
oF added testing time. But 
Mulhauser noted that the true 
cost depends on the purpose For 
such a test, and he urged Con- 
gress to be clear about the pur- 
pose be Fore determining the pro- 
posed cost. 

Similarly, Lawrence Picus, 
CRESST/Universir\' oFSouthern 
California, also raised a number 
oF questions that need to be an- 
swered before determining 
whether the benefits of perfor- 
mance assessment exceed the 
costs. Am( »ng these questions are : 



whether we want to compare in- 
dividual students or school dis- 
tricts, how many tasks are needed 
to provide a reliable estimate of 
student abilities, how much train- 
ing is needed for teachers, and 
what are the competing claims for 
resources. 

As in other issues of public 
policy, Picus said, deciding 
whether to invest in performance 
assessment involves tradeoffs: 
"How much of this do you want 
to do versus how much can you 
attbrd to do in terms of time and 
resources.^" 

Equity- and Assessment Design 
In looking ahead to new pro- 
grams, researchers also discussed 
possible ways to ensure equir/ in 
assessment. 



Equity is one of the 
key criteria by which 
educators can judge 
tests. 



One step toward that end is 
being taken by the National Coun- 
cil of Teachers of Mathematics 
(NCTM). In that group's Assess- 
ment Standards far School Math- 
cmaticsy equit\' is one of the key 
criteria by which educators can 
judge tests, according to Thomas 



A. Romberg of the Universit}^ of 
Wisconsin, Madison. Romberg 
said that the NCTM defines eq- 
uity as providing all students the 
opportunit}^ to demonstrate their 
mathematical power, and he noted 
that current tests do not match 
the council's goals. 



"...[Native Ameri- 
cans] also value pa- 
tience, whereas tests 
demand rapid re- 
sponses and immedi- 
ate decisions." 



Other researchers suggested 
that new assessments must take 
into account the needs of diverse 
students in order to be equitable. 
Michael Pavel of the UCLA 
G rad ua t e Sch ool of Ed ucation said 
that Native American students are 
ill-ser\'ed by traditional tests. Na- 
tive Americans value placidity, a 
characterisric that may result in 
their being viewed as slow or back- 
ward, he said. Similarly, patience 
is valued, whereas tests demand 
rapid responses and immediate 
decisions. 

He added that assessment 
should be better adapted to Na- 
tive American people who need 
to improve on their academic per- 
formance and make teachers more 
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aware of how rhey can address 
these academic needs. Therefore, 
to assist Native Americans, assess- 
ment results should be used to 
guide student learning, refine cur- 
riculum, and improve instruction. 

Charlene Rivera of George 
Washington Universit\- said as- 
sessment reform must also con- 
sider students for whom English 
is not the native language. In ad- 
dition to learning content knowl- 
edge and skills, she pointed out, 
'^English language learners" 
( ELLS ) are also learning a second 
language, something native En- 
glish speakers do not have to do. 

"To date, reform effons have 
not considered needs of ELLS 
students,*' Riverasaid.''Theguid- 
ing assumptions have been: What 
will work for monolingual stu- 
dents will also work for ELLS 
students. Once ELLS students 
learn a little English, new im- 
proved assessment systems will fit 
them too. However, experience 
doesn*t support this assumption. 
While ELLS students can and do 
learn to high standards, assessing 
their achievements in the same 
way as their monolingual peers 
will greatly underestimate their 
accomplishments and potential." 

Rivera added that schools cur- 
rently include English language 
learners in inappropriate testing 



programs or 'Ise exempt them 
ft-om tests altogether. am on 
the side of trying to develop a 
middle ground where there is ac- 
countability' for student learning," 
she said. "If you exclude them 
completely, they are not consid- 
ered in the policy debate. But 
how to assess them? There is no 
definitive answer. The best prac- 
tices will result from experimen- 
tation." 



Experiments already 
underway suggest pos- 
sible solutions for tra- 
ditionally underserved 
groups. 



Some experiments already un- 
der way suggest possible solu- 
tions for English language learn- 
ers and other traditionally 
underserved groups. One such 
experiment is a primar\'-grades 
assessment currendy being devel- 
oped by a consortium of six states 
under the auspices of the Council 
of Chief State School Officers. 
Jackie Cheong of the University 
of California, Davis, said one of 
the guiding principles of that ef- 
fort is to provide multiple assess- 
ment strategies to tap a range of 
what students from diverse back- 
grounds know and are able to do. 



Establishing the link between 
classroom assessments, such as 
portfolios, and accountability is 
one wa>' to further the use of 
multiple assessments. 

Porttblios 

Likewise, portfolios can also 
provide diverse opportunities for 
students to demonstra^;! their skills 
and knowledge. A school project 
in Pasadena, California, that inte- 
grates language arts and visual 
and performing arts instruction 
by asking students to create works 
of art and to reflect on their own 
and other works has shown dra- 
matic results among a group of 
formeriy low-achieWng children, 
according to Pam Aschbacher, 
CRESST/UCLA. 

But x'Kschbacher also pointed 
out that implementing portfolios 
alone may not improve educa- 
tional opponunities for all youths 
and the portfolios themselves can 
demonstrate this. In a separate 
project, Aschbacher examined 
portfolios to open a window on 
classroom instruction and found 
that teachers were not always 
meeting reform objectives. Un- 
der the program, known as 
Humanitas^ students were ex- 
pected to be personally invested 
in the work that they're doing in 
class, to make interdisciplinary 
connections, to engage in com- 
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picx reasoning, and to evaluate 
themselves and show some growth 
over time, among other goals. 

The portfolios showed a direct 
relation bet\vcen the kind of as- 
signments teachers gave their stu- 
dents and what students learned. 
For example, she found that stu- 
dents' work showed more higher 
order thinking skills and interdis- 
ciplinars' connections when their 
class assignments required them 
to make those connections and to 
use complex thinking. "That's a 
strong message," smd Aschbachcr, 
"to tliese teachers that says: * Look, 
when you use assignments that 
do not explicitly ask kids to do the 
kind of thinki ng this program calls 
for, they don't do it. You get 
what you ask tor/" 

Similarly, Catherine Smith of the 
Michigan Department of Educa- 
tion said tliat a program in that 
state to use portfolios to gauge 
students' workforce readiness also 
raised equit\' concerns. "Students 
in inner-cit\' and rural areas were 
less likely than those in suburbs to 
be aware that they could use evi- 
dence from part-time jobs and other 
afterschool activities to demonstrate 
readiness skills," she said. 

Maryl Gearhart, CRESST/ 
UCLA, also reported that imple- 
menting portfolio assessment 
alone may not have much impact 
on classroom practice. Describ- 



ing nvo R&D projects on class- 
room assessment, one focused on 
^^xiting and the other on math- 
ematics, Gearhart said, "I have 
found that elementary teachers 
do not typically develop in their 
understandings of student work 
without a substantive focus on 
subject matter. New assessments 
require teachers to make informed 
judgments," she said, "^but teach- 
ers cannot judge work that they 
do not understand. In our 
projects, the goal is to create pro- 
totypes of assessment practices 
iHtcjjrated with reform curricula. 
VVc can't simply exhort teachers 
to collect student work and assess 
it," said Gearhart. "We need to 
give them specific models built 
upon specific curriculum." 

Group Assessment 

Similar caution flags went up 
over another potential avenue for 
ensuring equity: the use of group 
work. Wayne Neuberger of the 
Oregon Department of Educa- 
tion suggested that allowing stu- 
dents to W'ork in groups in ad- 
vance of an assessment could level 
the playinjf field by enabling stu- 
dents to compare notes and en- 
sure that they all had the same 
background to prepare for the 
assessment. The New Standards 
Project, of which Oregon is a 



member, provides an opportu- 
nity to do just that. 

But Noreen Webb, CRESST/ 
UCLA, noted that group inter- 
actions are complex and that sim- 
ply asking students to cooperate 
may not result in everyone's ac- 
quiring shared knowledge and 
understandings. Factors such as 
the composition of the group 
and the incentives they have for 
working together can intluence 
whether group interactions are 
functional or dysfunctional, 
Webb said. 

Moreover, she added, relying 
on group assessments as a mea- 
sure of student abilities may ad- 
versely aftcct equit\'. Depending 
on the way groups work together, 
she said, group performances may 
not be a valid measure of the 
performance of individual mem- 
bers, and, as a result, group per- 
formances may mask individual 
deficiencies. If that is the case, 
Webb warned, low pertbrmers 
may miss out on needed instruc- 
tional help. 

Equity and Interpreting As- 
sessment Results 

In addition to the design of 
new assessments, the way results 
are interpreted also has a bearing 
on questions of equity. Da\id 
Bayless of Bayless and Associates 
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said that the traditional way of 
analyzing assessment results is to 
look at subgroups, such as gen- 
der, socioeconomic status, and 
ethnicity. But those analyses can 
provide little guidance tor im- 
provement, because they cannot 
be changed. "It's difficult to 
change (a student's] gender," he 
said. RAther than select demo- 
graphic subgroups that are easy to 
measure. Bay less argued, analysts 
should examine assessment results 
according to factors that can be 
inrtuenccd by inter\'entic)n, such 
as opportunit}' to learn. 



Examining single test 
scores may mask im- 
portant differences in 
subgroups' perfor- 
mance. 



Bengt Muthen, CRESST/ 
UGL-A, said that examining single 
test scores may mask important 
differences in subgroups' pertbr- 
mance. In one study, Muthen said 
he is using multivariate analysis to 
gain a broader picture of student 
pertbrmance on the National As- 
sessment of Educarional Progress. 
Pre!iminar\^ findings suggest that 
while female and male students 
perform about equally well over- 
all, male students outperform fe- 



males on certain subscales, such 
as measurement. Muthen is also 
analyzing the results of the Lon- 
gitudinal Study of American 
Youth to detect the cffectsoftrack- 
ing and coursetaking on student 
performance over time. 

Looking at a separate way of 
exaniiningpossiblesourcesofbias, 
Jamal Abedi of CRESST/UCLA 
said that different methods of 
analyzing inverraterreliabilit}' may 
yield difierent estimates. As a re- 
sult, he suggested using different 
applicable approaches. "I cannot 
name a single best approach to 
establishing in terraterreliabilit\%" 
Abedi said. "Depending on the 
form of data, compute as many 
applicable approaches as you can, 
and then draw your conclusions 
based on those outcomes." 

Michael T. Nettles of the Uni- 
versit\* of Michigan proposed an 
additional step. After citing eth- 
nic group differences in pertbr- 
mance on various assessments, he 
said that we should look at assess- 
ments that show good results with 
traditionally low-performing stu- 
dents. 

Under the auspices of the Ford 
Foundation, Nettles said that 
during the next year he expects to 
develop a second symposium on 
equi t\' and educational testing and 
assessment and develop a pro- 



gram to identiH' exemplary as- 
sessment programs. One of the 
criteria for being considered ex- 
emplary should be that the assess- 
ments have demonstrated mak- 
ing a contribution to improving 
the success of poor and minority 
student populations. 

Equity and the Research Agenda 
Although some answers to the 
equityquestionssurroundingnew 
forms of assessment are begin- 
ning to emerge, the research 
agenda remains long. Pauline 
Brooks, CRESST/UCLA, out- 
lined a host of questions in light 
of the unequal representation of 
gender, socioeconomicstatus, and 
diverse cultures throughout the 
histoty of tesring and assessment. 
Included on her list were: 

• the effects of the teacher's 
cultural expectations on 
his/her judgments of cul- 
turally diverse student per- 
formances; 

• the extent to which the new 
assessments provide vaty- 
ing opportunities for stu- 
dents of different cultural/ 
SES/language backgrounds 
to demonstrate their 
Icnowledge; 

• the correlation between 
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pcrtbrmancc on standard- 
ized achievement tests and 
current pertbrmance assess- 
ments — do the racial/cul- 
tural and SES gaps narrow, 
remain about the same, or 
widen with the use of per- 
formance assessments? 

• the characteristics of teach- 
ers' interactions with dif- 
ferent cultural groups of 
students, in both instruc- 
tional and assessment set- 
tings; and 

• relative levels of support for 
the new assessments among 
communities that wm eco- 
nomically and culturally. 

Working Group Results 

In addition, working groups 
representing various key constitu- 
encies outlined action plans for 
designing equit)'-sensitivc perfor- 
mance assessments. The groups 
recommended the following: 

Policy Makers: 

• Clearly articulate the pur- 
poses of new assessments, 
so that the public under- 
stands that assessments arc 
aimed at meaningful and 
effective accountability', not 
an attempt to evade account- 
abilit\'. 



• Make sure assessment for 
''classroom utilit\'" and as- 
sessment for public ac- 
countabilirv are in sync, so 
that teachers have an incen- 
tive to focus on improved 
instruction. 

• Keep expectations high for 
all students to eliminate any 
incentive to relegate low- 
performing students to a 
second-class education. 

Practitioners: 

• Make the purpOsSes and 
learning outcomes of new 
assessments clear and make 
sure that large-scale and 
classroom-level assessments 
are integrated. 

• Encourage tlexibilit)'so that 
all students have the time 
and opportunities to suc- 
ceed on new assessments. 

• Provide choices in assess- 
ment alternatives to chil- 
dren that are appropriate to 
their ethnicit\\ their gen- 
der, the possible existence 
of handicap, the language 
that they use, etc. Incorpo- 
rate diverse groups during 
the design and piloting pro- 
cess so that we engage all 



CO m m uni tie s and cons titu - 
encics. 

• Develop university-school 
partnerships to share knowl - 
edge about children's learn - 
ing and provide ongoing 
staff development for teach- 
ers. 

• Conduct longitudinal and 
comparative studies to ex- 
amine the impact of new 
assessment strategies on 
teachers and students. 

Foundations: 

• Make sure new programs 
are practical and feasible 
so that they can be imple- 
mented in schools and 
yield usefol findings. 

• Make sure programs are 
credible to parents, teach- 
ers, and tiinding agencies 
and that the programs are 
actually implemented. 

• Tr\' to focus on the "hard- 
to-crack" cases. 

Business Community: 

• Educators, businesses, and 
the communit\^ at large 
need to find a better way 
to collaborate to produce 
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diagnostic assessments that 
enable our students to ad- 
vance their own goals in the 
classroom and in the work- 
place. 

• Assessments must also be 
predictive and show stu- 
dents' readiness for the 
workplace as well as their 
ability to transfer skills from 
school to work. 

• Considerwhcther all assess- 
ments must be administered 
in school or by teachers or 
whether other approaches 
are possible, depending on 
the purpose. 

Media: 

• Highlight the standards and 
content of new assessments 
as a way of showing what all 
students are expected to 
learn, 

• Report opportunity to leam 
data and classroom envi- 
ronments to provide a bet- 
ter understanding of eq- 
uity. 

• Develop links between re- 
searchers and the media to 
provide context for test- 
score data. 



Parents: 

• Involve parents in all as- 
pects of the development 
of new assessments via ad- 
vocates or site-based man- 
agement, helping to ensure 
that they are fidr and better 
for all children. 

• Collect data on instruction, 
testing participation, growth 
in student performance, and 
teacher grading, to accom- 
pany test-score data. 

• Conduct research on par- 
ent involvement, toxic 
schools^ the role of churches, 
and communication of test- 
ing information. 

In Conclusion 

In concluding remarks, Adam 
Urbanski, the president of the 
Rochester (NY) Teachers Asso- 
ciation, pointed out that reform- 
ing schools so that all students 
learn at high levels is a long and 
difficult process that must involve 
the entire community. But he said 
that setting standards and devel- 
oping new assessments is the es- 
sential starting point of such ef- 
forts in order to ensure bcth ex- 
cellence and equity. 

"I respectfully suggest that all 
students are learning already,'' said 



Urbanski. "TheyVe just not learn- 
ing the same things. Some stu- 
dents are learning math and En- 
glish and foreign languages and 
the arts and physics. And some are 
learning a lot about exclusion and 
tailure and discrimination and lack 
of opporranities and low expecta- 
tions of them. But you can't stop 
students from learning. You can 
only assist in channeling them to 
certain 'kinds' of learnings. That 
is why the whole issue of stan- 
dards and assessment is so ger- 
mane. They are absolutely essen- 
tial and not only essential, but 
indeed the necessary starting point 
for all other reforms. School 
change need not be a choice be- 
tween making things better or 
making things fair. We Ve capable 
of doing both. We must make 
things better and be fair in doing 
it." 

Robert Rothman is a visiting 
researcher at CRESST/ UCLA, 
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:< VERMONT UPDATE 
Can Portfolios Assess Student 
Performance and Influence In- 
struction? The 1991-92 Ver- 
mont Experience 
Daniel Koretz, Brian Stcchvr, 
Stephen Klein, Daniel McCaffrey, 
and Edward Deibert, RAND 
CSE Technical Report 371, 199.3 
($9.00) 

Vermont *s statewide assessment 
initiative program has garnered 
widespread attention nationwide 
because of its reliance on portfo- 
lios of student work. This 145- 
page CRESST/RAND report de- 
scribes results of a multifaceted 
evaluation of the program and 
provides information about imple- 
mentation of the Vermont assess- 
ment, program effects on educa- 
tional practice, reliability and va- 
lidity of portfolio scores, and ten- 
sions that exist between assess- 
ment and instructional reform. 

"Findings from the evaluation,'' 
said the research team, "suggest 
that the assessment program re- 
sulted in changes in curriculum 
content and instructional style.'' 
Additionally, the researchers 
noted that the amount of class- 
room time devoted to problem 

Jt NEW REPORT 



solving increased, as did the 
amount of time students worked 
in small groups. Finally, portfo- 
lios seem to increase teachers' 
enthusiasm for their subjects and 
for teaching. 

While there was widespread 
support for the reform at the 
school level throughout the 
state — nearly one-half of the 
schools were voluntarily expand- 
ing the use of portfolios to other 
grade levels — substantial prob- 
lems remain. The mathematics 
portfolio assessment created new 
burdens for principals, teachers 
and students, including demands 
on teachers' time and school re- 
sources. Over 80% of fourth- 
grade teachers and over 60% of 
eighth-grade teachers reported 
that they often had difficulty cov- 
ering the required curriculum. 
Researchers anticipate that in time 
some of these demands are likely 
to decline, although others repre- 
sent continuing burdens. 

"The Vermont experience has 
important implications for reforms 
that are underway or under con- 
sideration in other jurisdictions," 
said the researchers, "but only 
time and careful scrutinv will show 



how^ fully the goals of the pro- 
gram — and of similar reform pro- 
grams centered on pertbrmance 
assessment — can be met." 

.V MORE ABOUT VERMONT 
Interim Report: The Reliabil- 
ity of Vermont Portfolio Scores 
in the 1992-93 School Year 

Daniel KoretZy Stephen Klein, 
Daniel McCaffreyy and Brian 
Stecher, RAND 

CSE Technical Report 370, 1993 
($3.00) 

This interim report provides the 
first results of the second year of 
the Vermont assessment program, 
focusing on program implemen- 
tation, effects on education, and 
the quality of performance data. 

"The program was altered in 
many wa\^ in 1992-93," said the 
CRESST/RAND research team, 
"which resulted in a clear increase 
in the reliability with which math- 
ematics portfolios were scored." 
However, the researchers added 
that while this progress is encour- 
aging, scoring reliability in math- 
ematics needs to be increased fur- 
ther if the program goals are to be 
achieved. Refining or simplifying 
scoring rubrics and placing fur- 
ther restrictions on types of tasks 
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considered acceptable for inclu- 
sion in mathematics portfolios are 
among the types of clarifications 
that may result in increased reli- 
ability. 

"^In contrast,'" said the research- 
ers, "^the reliability of \\Titing port- 
folio scores did not improve sub- 
stantially and was considerably 
lower than in mathematics." The 
researchers believe that it is unre- 
alistic to expect a substantial rate 
of improvement in the reliability 
of the writing portfolio scores 
unless the program itself is sub- 
stantially revised, 

.V Comparability Across Assess- 
ments: Lessons From the Use 
of Moderation Procedures in 
England 

Elizabeth Bttrton and Robert L 
Linn 

CSE Technical Repon 369, 1994 
($4.00) 

Although there is considerable 
interest in developing a system of 
performance- based examinations 
in the United States, there is a 
general lack of agreement on how 
to compare the results of different 
performance assessments to sets 
of common national standards. 
This paper addresses the **com- 
parison"' problem, drawingon two 
major approaches used in En- 
gland, moderation by inspection 



and stf.itistical moderation, to link 
performance assessments to sets 
of common standards. 

Although the United States will 
probably not develop a program 
of assessments exactly like those 
used in England, it is likely that 
the procedures used to compare 
the assessments will be similar. 

"Currently English secondary 
school exams in various subjects 
are developed and administered 
by nine examination boards,'' said 
Burton and Linn. "Individual 
schools are tree to choose the 
examination board that best fits 
their standards. While local con- 
trol and high quality of assess- 
ments are maintained, the com- 
parison of scores across the boards 
is problematic," added the re- 
searchers. 

In this report. Burton and Linn 
discuss the advantages and prob- 
lems of moderation by inspection 
and statistical moderation^ to- 
gether wdth an explanation of why 
neither approach is satisfactory by 
itself The authors concluded that 
some combination of the two 
approaches may be necessary, 
"Neither a pure moderation by 
inspection nor a strict statistical 
moderation system is likely to meet 
this [link between assessments and 
standards 1 need,** said Burton and 
Linn. "It seems more likely that 



some sort of hybrid system will be 
required..." 

.V Results From the New Stan- 
dards Project Big Sky Scoring 
Conference 

Lauren Resnicky Daniel Resnicky 
and Lizanne DcStefano 
CSE Technical Report 368, 1993 
(S3,50) 

Partially ftinded by CRESST, 
the New Standards Project is an 
effort to create a state- and dis- 
trict-based assessment and pro- 
fessional development system that 
will serve as a catalyst for major 
educational reform. In 1992, as 
part of a professional develop- 
ment strateg}' tied to assessment, 
1 14teachers,curriculum supervi- 
sors, and assessment directors met 
to score student responses from a 
field test of mathematics and En- 
glish language arts assessment. 
The results of that meeting, the 
Big Sky Scoring Conference, were 
used to analyze for comparability 
across holistic and anaholistic scor- 
ing methods. 

"Interscorer reliability esti- 
mates," said the researchers, "for 
reading and writing were in the 
moderate range, below levels 
achieved with the use of large- 
scale writing assessment or stan- 
dardized tasks. Lx)w reliability lim- 
its the use of [the] 1992 reading 
and writing scores for making 
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judgments about studcnc perfor- 
mance or educational programs/' 
concluded the research team. 

However, interscorer reliabilit\' 
estimates tor math tasks were 
somewhat higher than tor literacy. 
For six out of seven math tasks, 
reliability coetiicients approached 
or exceeded acceptable levels. 

The tindings suggest that the 
large number and varied nature of 
participants may have jeopardized 
the production of valid and reli- 
able data. "Scorers reported feel- 
ing overwhelmed and overworked 
atter tour days of training and 
scoring," said the researchers. 

Despite these ditficulties, evi- 
dence was provided that reliable 
scoring of large -scale pertbrmance 
assessments can be achieved when 
ample time is provided for train- 
ing, evaluation, feedback, and dis- 
cussion; clear definitions are given 
of pertbrmance levels and the dis- 
tinctions between them; and well- 
chosen exemplars are used. 

> Parent Opinions About Stan- 
dardized Tests, Teacher's In- 
formation and Performance 
Assessments 

Lorrie A. Shepard and Carribeth 
L. Bliem 

CSE Technical Report 367, 1993 
($4.00) 

Using parents of third-grade 
students in a working-class and 



lower-middle-class school district, 
researchers . .. .his study set forth 
to ascertain parents' opinions 
about assessment, including their 
opinions about standardized tests 
versus pertbrmance assessments. 
The researchers sought answers 
to several questions including 
*'How do parents in the sample 
respond to Gallup Poll questions 
about the desirability' of standard- 
ized national tests and the poten- 
tial uses tor standardized test re- 
sults?" As part of the study, par- 
ents were given an opportunity to 
re\iew pertbrmance assessment 
tasks and decide what t\pe of 
assessment, standardized or per- 
tbrmance, was most suitable for 
classroom use. 

The results indicated that when 
allowed to look closely at pertbr- 
mance assessment problems, most 
parents endorsed performance 
assessments for district purposes 
and especially preferred their use 
in classroom contexts. 

Teachers' Ideas and Prac- 
tices About Assessment and In- 
struction 

Hilda Borko, Manrcne Flo}y, and 
KateCumbo 

CSE Technical Report 366, 1993 
($4.00) 

Participants involved in this 
study were part of a year-long 



intervention designed to help 
teachers develop pertbrmance as- 
sessments in reading and math- 
ematics. Seeking to evaluate teach- 
ers' knowledge, beliefs, and prac- 
tices about assessment and in- 
struction, the researchers also 
studied the changes that occurred 
to teachers during the tirst semes- 
ter of the inter\'ention program. 

Findings trom the study indi- 
cated that the pertbrmance as- 
sessment development and imple- 
mentation process resulted in 
teachers having better under- 
standings and new insights into 
students' thinking and learning 
than when teachers relied exclu- 
sively on more traditional forms 
of assessment. However, it was 
not clear to what extent teachers 
changed their instructional pro- 
grams to take advantage of their 
newly gained insights. Based on 
their observations so far, research- 
ers feel confident that as the pro- 
gram continues, more extensive 
changes will occur. 

'V Dilemmas and Issues in 
Implementing Classroom- 
Based Assessments for Literacy 
Eljrieda H. Hiebertand Kathtyn 
Davinroy 

CSE Technical Report 365, 1993 
(S3. 50) 

Researchers in this study in- 
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\ated third-grade teachers from 
an urban school district to col- 
laborate in a classroom-based lit- 
eracy assessment project. The 
study focused on a series of lit- 
eracy workshops designed to 
implement a long-standing per- 
spective on curriculum, instruc- 
tion and assessment adapted to 
classroom -based assessment. 

Some of the early outcomes 
from observations and transcrip- 
tions of the workshops indicated 
that teachers struggled with a va- 
riet)' of issues including the task of 
embedding assessments like run- 
ning records and written summa- 
ries into their instructional pro- 
grams. Despite many challenges, 
at least one of the schools moved 
quickly to implement the assess- 
ments and use the information 
the assessments provided. 

.V Dilemmas and Issues for 
Teachers Developing Perfor- 
mance Assessments in Math- 
ematics 

Roberta /. Flexer nnd Eileen A, 
Gerstner 

CSE Technical Report 364, 1993 
($4.00) 

This paper examines some of 
the dilemmas and issues that arose 
during the first two terms of work 
with teachers participating in the 
development of assessments in 



mathematics, and reports on 
changes in their instruction and 
assessment as a result of the 
project. 

During the study, many dilem- 
mas and issues arose that were 
unique to each of the three schools 
studied, but the most challenging 
problem was teachers' focus on 
what was important to teach (and 
therefore assess), and how chil- 
dren cou)d learn what was 
taught — all within the constraints 
of limited teacher time. As ex- 
pected, Preliminary results of the 
project were mixed, but hopejflil. 
Researchers believe that ftiture 
development and implementation 
of performance assessments in 
these classrooms hinge on teach- 
ers' belieft in these assessments as 
usetiil and practical tools. 

Whose Work Is It? A vjucstion 
for the Validity of J^ge-Scale 
Portfolio Assessment 
MarylGearhanJoan L. Hennariy 
Eva L Baker, and Andrea K. 
Whittaktr 

CSE Technical Report 363, 1993 
($3.00) 

Portfolio assessment represents 
a growing commitment to bridge 
the worlds of public accountabil- 
ity' and private classroom, anH 
policy maker and child. Thus, 
within the move toward further 



authenticity, portfolios support 
performance-based assessments 
that may incorporate shared read- 
ings of common background 
texts, collaborative planning, and 
opportunities fo- students to re- 
vise their work. 

Based on an in-depth analysis 
of nine elementary school teach- 
ers actively using writing portfo- 
lios in their classrooms, the re- 
searchers of this study focused on 
a technical issue not yet directiy 
investigated in R&D studies of 
portfolio assessment: ^W!wsework 
isit?^l( students collaborate with 
peers or receive assistance from 
parents or teachers, the author- 
ship of classroom work of the 
student is in question. Focusing 
here just on the teachers' contri- 
butions to student work, the au« 
thors documented patterns of in- 
structional support across writing 
assignments and students. The 
work raises technical issues con- 
cerning the meaningfulness of 
'student' scores derived from as- 
sessment of student portfolios. 
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Performance- Based Assessment 
and What Teachers Need 

Huftichi 

CSE Technical Report 362, 1993 
($4.00) 
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